Search AI Products and News

In geoscience, geological maps are crucial tools for understanding the Earth's surface and subsurface structures. However, interpreting these complex diagrams requires specialized knowledge and extensive experience. To enhance intelligence in this field, Microsoft Research Asia recently introduced GeoMap-Bench, a new benchmark dataset for evaluating the performance of multimodal large language models (MLLMs) in understanding geological maps. The launch of GeoMap-Bench marks a significant step forward in AI applications for geological map interpretation. Microsoft researchers, in collaboration with...

Ovis2 is the latest version of the Ovis series models proposed by Alibaba's international team. Compared to the previous version 1.6, Ovis2 has significant improvements in data construction and training methods. It not only enhances the capacity density of small models but also greatly improves chain of thought (CoT) reasoning capabilities through instruction fine-tuning and preference learning. Additionally, Ovis2 introduces video and multi-image processing capabilities, and enhances multilingual abilities and OCR capabilities in complex scenarios, significantly increasing the model's practicality.

Driven by multimodal large language models (MLLMs), significant advancements have been made in tasks related to images and videos, including visual question answering, narrative generation, and interactive editing. However, achieving fine-grained understanding of video content still poses major challenges. These challenges involve tasks such as pixel-level segmentation, tracking with language descriptions, and visual question answering based on specific video prompts. Although current state-of-the-art video perception models excel in segmentation and tracking tasks, they still fall short in open language understanding and conversational capabilities.

Recently, the Meta AI team introduced LongVU, a novel spatio-temporal adaptive compression mechanism aimed at enhancing the language understanding capabilities of long videos. Traditional multimodal large language models (MLLMs) face limitations in context length when processing long videos, and LongVU was created to address this challenge. LongVU operates primarily by filtering duplicate frames and employing inter-frame token compression techniques to efficiently utilize context length, allowing it to reduce video data while preserving visual details.

Recently, Apple's AI research team launched their next-generation family of Multimodal Large Language Models (MLLMs) - MM1.5. This series of models can integrate various data types such as text and images, showcasing new capabilities of AI in understanding complex tasks. Tasks like visual question answering, image generation, and multimodal data interpretation can be better addressed with the help of these models. A major challenge for multimodal models is how to achieve effective interaction between different data types. Previous models often struggled in this aspect.

As AI technology continues to advance, understanding user interfaces (UI) has become a key challenge in creating intuitive and useful AI applications. Recently, researchers at Apple introduced UI-JEPA in a new paper, an architecture designed for lightweight, device-side UI understanding that maintains high performance while significantly reducing the computational requirements for UI understanding. The challenge of UI understanding lies in processing cross-modal features, including images and natural language, to capture temporal relationships within UI sequences. Despite the complexity, multimodal large language models...

NVIDIA has collaborated with several universities to introduce NVEagle, a large visual language model capable of chatting using images. NVEagle can analyze image content and provide accurate answers, such as identifying individuals in images, like Jensen Huang. The model significantly enhances the understanding of visual information by transforming images into visual tokens and combining them with text embeddings. In addressing the challenges of high-resolution image processing, the research team has constructed models like Eagle-X5-7B and Eagle-X by exploring various visual encoders and fusion strategies.

Tencent's YouTu Lab and other institutions have released the first open source multimodal large language model VITA, aimed at bridging the gap in processing Chinese dialects. Based on the Mixtral8×7B model, VITA expands the Chinese vocabulary and undergoes bilingual instruction fine-tuning, mastering both English and Chinese. Key features include: 1. **Multimodal Understanding**: VITA can handle video, images, text, and audio, which is unprecedented among open source models. 2. **Natural Interaction**: No specific wake words are required, allowing for instant response while maintaining polite and non-intrusive communication.

Recently, an open-source project called gptpdf has gained 1.1k stars on GitHub. It uses a VLLM model similar to GPT-4o to parse PDF files and convert them into Markdown format.gptpdf Product Entry: https://top.aibase.com/tool/gptpdf It is learned that the code for this project consists of only 293 lines, yet it can nearly perfectly parse and include content such as formatting, mathematical formulas, tables, images, charts, and more. The implementation steps of gptpdf are: 1) Use the PyMuPDF libr